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Take  Aways 


COMOPTEVFOR  leveraged  a  DEP  test  event  to 
provide  data  supporting  joint  interoperability 
assessment  in  an  OA 

Interoperability  metrics  cued  root-cause  discovery 
of  a  serious  interoperability  issue 

NGC2P  program  pursuing  TTP  and  materiel  risk 
mitigation  efforts  prior  to  OPEVAL 

DEP  is  a  useful  tool  supporting  interoperability 
testing  of  major  combat  systems  and  their  ancillary 
equipment  in  a  Joint  environment 

An  OT  resource,  especially  in  early  phases  of 
testing 


Overview 


■**  Leveraging  of  DEP  event 

•  VV&A  lessons 

•  DEP-based  assessment  lessons 


What’s  a  DEP? 


Distributed  Engineering  Plant  (DEP) 

Hardware  in  the  loop  (HWIL) 

Hardware  =  entire  major  combat  systems 
and  ancillary  equipment 

Multiple  system  labs  linked  by  ATM-VLAN 

Sensors  stimulated  with  common  scenario 

TDL  connectivity  via  GTE  over  ATM-VLAN 


OA  System  Under  Test:  NGC2P 


•  Next  Generation  Command  and  Control 
Processor  (NGC2P) 

-  Surface  Navy  tactical  data  link  (TDL)  interface 

-  New  increment  adds  JRE  and  other  capabilities 

•  NGC2P  OA  in  Match  06 

-  Hawaii  and  SOCAL 

-  Two  Aegis  ships  and  land-based  test  site 

-  Brief  participation  in  strike  group  TDL 

-  Non-Navy  participants  not  avaiiabie  for  test 
to  support  assessment  of  joint 

in  teroperabiiity 


C/DIT  06  Joint  DEP  Event 


Correlation/Decorrelation  Interoperability  Test  (C/DIT) 
runs  for  record  (first  week)  in  DEP  27  Feb  -  2  Mar  06 

Ship  Self  Defense  System  (SSDS)  participant  uses 
NGC2P  as  TDL  interface 

Many  other  Navy  and  joint  participants 

-  Navy:  Aegis  (x2),  ACDS  Block  1,  E-2C  HE2K 

-  Joint:  Army  Patriot  and  Air  Force  E-3  (AWACS) 

Opportunity  to  expand  the  scope  of  the  OA  to  better 
address  joint  interoperability 

COMOPTEVFOR  obtained  permission  of  NGC2P 
program,  NAVSEA,  and  C/DIT  director  to  observe  and 
analyze  data  in  addition  to  live  NGC2P  OA  events 
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C/DIT  06  DEP  Event  Architecture 
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C/DIT  06  Adjunct  to  the  NGC2P  OA 


•  C/DIT  Primary  objective  to  test  MILSTD 
601 6C  track  correlation  protocol 
(Corr/Decorr  ICP) 

•  But  baseline  cases  were  run  “without” 
Corr/Decorr  ICP  (tactical  loads) 

•  Robust/challenging  71 -object  scenario 
largely  derived  from  live  Red  Flag  and  JCIEl 
events 

•  Scenario  and  diversity  of  participants  ideal 
for  OA  assessment  of  joint  interoperability 
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Efficiencies 


•  C/DIT  analysis  team  performed  air  picture 
reconstruction  supporting  primary  C/DIT  test 
objectives 

•  C/DIT  data  reduction  and  reconstruction  a 
fully-leveraged  input  to  NGC2P  joint 
interoperability  assessment  analysis 

•  OT&E  cost  limited  to  observer  TAD  and 
analyst  time  (-3  months)  for  V&V  and  data 
analysis 
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Overview 


•  Leveraging  of  DEP  event 
->•  VV&A  lessons 

•  DEP-based  assessment  lessons 
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Verification,  Vaiidation  & 
Accreditation  (W&A) 


•  COMOPTEVFOR  Policy  requires  M&S  W&A 
for  HWIL 

•  Inadequate  time  to  perform  V&V  prior  to 
event 

•  Opted  for  concurrent  V&V  and  analysis 

•  Accreditation  and  V&V  plans  developed  per 
policy 

-  M&S  requirements,  assessment  methods, 
acceptance  criteria: 

-  Stimulation  to  NGC2P  host  combat  system  and 
other  C/DIT  participant’s  sensors 

-  Emulation  of  TDL 
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W&A  Approach 


•  Review  relevant  prior  DEP  V&V  efforts 

•  Review  results  of  C/DIT  integration, 
risk-reduction,  and  dry-runs 

•  COMOPTEVFOR  observations  at 
ICSTD  during  the  test 

•  C/DIT  team  observations  at  other  sites 

•  Post-test  analysis  of  combat-system 
track-file  and  data  link  extracts 
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V&V  Findings 


•  Six  of  eleven  requirements  completely 
satisfied 

-NGC2P  host  interface 

-Support  of  multiple  combat  systems 

-DIS  interfaces 

-Scenario 

-IFF  data  errors 

-Airborne  participants  in  DIS;  consistent  with 
TDL  data 
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V&V  Findings  (cntd.) 


•  Four  requirements  partially  satisfied 


-  Scenario  objects  within  coverage  tracked  locally 
•  Lack  AWACS  track  file  data  to  confirm  true  for  AWACS 


•  Aegis  5.3.8  participant  lacked  local  tracks  of  E-2  and  AWACS 

-  Scenario  consistency  across  units 

•  some  DIS  data  distortion  at  AWACS  causing  some  “track  jumps” 

•  IFF  data  anomaly;  later  determined  to  be  SGS/AC  issue  (not  DEP) 

-  Terminal  emulator  performance  (latencies  at  Patriot  site 
during  most  runs) 

-  Scenario  aircraft  non-C2  PPLIs  (update  rate  too  slow) 

•  CEC/DDS  requirement  not  satisfied  (no  data  from 
runs  where  DDS  was  used) 

•  Note:  most  of  the  plant-related  issues  from  C/DIT  06 

week  1  have  since  been  resolved  ^ 


Accreditation 


•  Plant  and  system  problems  precluded  meeting  many 
C/D  IT  06  objectives 

•  But  C/DIT  06  was  adequate  to  support  assessment  of 
NGC2P  joint  interoperability  for  the  OA  with  following 
restrictions: 

-  Portions  of  test  runs  with  terminal  emulator  latencies  were 
not  to  be  used 

-  Runs  where  DDS  operations  were  attempted  were  not  to 
be  used 

-  Runs  including  non-C2  PPLIs  were  not  to  be  used 

-  Effects  of  noted  anomalies  on  the  air  picture  to  be  carefully 
accounted  for  and  distinguished  from  NGC2P  performance 
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Parallel  V&V  and  Assessment 


•  V&V  and  assessment  based  on  identical  data 

•  Overlap  of  V&V  and  assessment  activities; 
efficiencies 

•  Analyst  experience/familiarity  with  strengths  and 
weaknesses  of  the  data  used  for  the  assessment 

-  Valuable  in  defending  results  and  obtaining  community 
acceptance  of  their  credibility 

-  DEP  data  withstood  considerable  post-report  scrutiny  due 
to  unanticipated  high-level  interest  in  the  findings 

-  Validity  of  findings  was  ultimately  acknowledged  by  all 
stakeholders 
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Overview 


•  Leveraging  of  DEP  event 

•  VV&A  lessons 

DEP-based  assessment  lessons 
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Interoperability  Assessment 

Methodology 

•  Operational  effects  of  interoperability  performance  manifest 
in  the  completeness,  clarity,  correctness,  and  commonality  of 
information  shared  among  FoS  operators 

•  Information/picture  quality  quantified  using  reconstructed 
operational  picture  attribute  metrics 

-  Each  system’s  tracks  matched  to  “truth”  tracks  (targets  instrumented) 

-  Attributes  scored  from  matched  tracks 

•  Picture  attributes  correspond  to  requirements/KPPs  in 
overarching  MA  ICDs 

•  Measures  end-to-end  operational  effectiveness  of  information 
exchanges  lAW  OTA  Commanders’  stated  NR-KPP 
assessment  role  (forest) 

•  Attribute  results  also  cue  root-cause  analysis  supporting  fault 
isolation  (trees) 
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Air  Picture  (SIAP)  Attributes 


•  Completeness:  %  of  objects  depicted 

•  Clarity:  %  of  depicted  objects  dualed 

•  Continuity*:  Track  number  changes/hour,  gaps 

•  Commonality**:  %  tracks  held  in  common 

(same  position+/-,  same  ID,  same  time) 

•  ID  Completeness:  %  objects  identified 

•  ID  Accuracy:  %  IDs  correct 

•  ID  Clarity:  %  IDs  unambiguous 

*”Rolled  up”  across  time  by  definition;  non-instantaneous 
**Rolled  up  across  units  by  definition 


21 


Operational  Picture  Attributes 


•  Broad  acceptance  of  SIAP  attributes 

-  Vetted  by  all  services 

-Widely  used  in  TAMD  communities 

•  KPPs  in  TAMD/CID  MA  ICDs;  required 
and  objective  performance  thresholds 

•  Reflected  in  recent  CDDs/TEMPs 

•  Adaptations  to  maritime  and  ground 
operational  pictures 
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Root-Cause  Analysis 


Analysis  of  attributes  cues  root-cause  analysis 

-  Reveals  generally  degraded  performance 

-  Reveals  performance  differences  between  units,  among 
reconstructed  objects,  vs.  time 

Frequently  spotlights  problems  not  observed  in 
real-time  or  during  initial  playback 

Cues  to  specific  tracks/times 

Focuses  analysis  to  relevant  system  data  to 
better  characterize  or  isolate  the  underling 
problem  or  problems 
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OA  Assessment 


•  Observed  degraded  commonality  attribute 

•  Drilled  down  to  specific  tracks/times 

•  Several  long-duration  uncommon  tracks 

•  Some  of  these  isolated  to  SSDS  host  (pair¬ 
wise  commonality) 

•  Examined  relevant  link  message  exchanges  in 
NGC2P  data 

-  For  affected  tracks  NGC2P  sent  different  track  number  to 
host  than  received  over  TDL 

-  Associated  with  earlier  receipt  of  AWACS  J7.4  messages 
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OA  Interoperability  Assessment 

Oiitr^nino 


Operational  Picture  attributes  cued  root  cause 
discovery  of  interoperability  problem 


-  NGC2P  interpretation  of  AWACS  J7.4  message 

-  Software  error  in  addition  to  the  TDL  standards  interpretation  issue 

-  Track  number  substitution  (passed  different  track  number  to  SSDS  host 
than  received  over  TDL) 

-  SSDS  saw  different  track  number  than  other  participants  on  same  track 

-  Occasionally  caused  track  numbers  to  be  applied  to  more  than  one  track 
at  a  time 

COMOPTEVFOR  reported  a  joint  interoperability  risk 
NGC2P  program  risk  mitigation  efforts: 

-  TTP/work-arounds,  coordinating  with  NCTSI,  NNWC,  and  USAF  (Global 
Cyberspace  Integration  Center  A6I) 

-  Software  correction 

DEP  use  in  OA  will  probably  result  in  better  outcome 
at  OPEVAL  later  this  year  than  would  have  occurred 
otherwise  ^ 


other  OA  Lessons/Considerations 


•  Learned  more  about  DEP  capabilities  and  limitations  in  OT 
context 

•  DEP  revealed  interoperability  issues  with  high  operational 
impact  across  the  family-of-systems 

-  NGC2P/AWACS  issue 

-  Also,  SGS/AC  software  problem;  fix  has  been  developed 

•  Not  all  bad  news',  many  aspects  of  correct  NGC2P  processing 
in  the  joint  environment  were  demonstrated 

•  Serendipity  was  valuable,  but  cannot  be  counted  upon;  in 
general  need  to  plan  to  utilize  this  resource  whether  on  a 
dedicated  or  NIB  basis 

•  Prudent  to  include  potential  use  of  DEP  in  TEMP  negotiations 

•  Need  also  to  ensure  adequate  data  reduction  and  analysis 
resources/expertise  on  tap 


26 


DEP  is  a  Valuable  T&E  Asset 


•  Testing  in  a  joint  environment;  it’s  here! 

•  Operational  mission  capability  impacts  of 
major  combat  system  interoperability 

•  Expanding  scope:  joint  and  coalition  (UK 
participation  in  C/DIT  07  this  summer) 

•  Full-up  combat  systems  include  all 
operator  interfaces;  potential  to  explore 
HMI,  etc. 
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